Skip to content

Granite4 2b & 3b support #496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Oct 16, 2025
Merged

Granite4 2b & 3b support #496

merged 12 commits into from
Oct 16, 2025

Conversation

@yannicks1
Copy link
Collaborator

@yannicks1 yannicks1 commented Oct 1, 2025 •
edited
Loading

Supporting granite4 2b & 3b (attention only).

fms support has been merged into main here: foundation-model-stack/foundation-model-stack#472

implemented it in a backward compatible way, such that we can get it into vllm-spyre main without relying on a fms version post above PR.

@github-actions
Copy link

github-actions bot commented Oct 1, 2025

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

@yannicks1
fix fmt
039c59a 
Signed-off-by: Yannick Schnider <[email protected]>
@yannicks1
load granite model types correctly too
e46ff48 
Signed-off-by: Yannick Schnider <[email protected]>
@yannicks1
remove print
f03d61b 
Signed-off-by: Yannick Schnider <[email protected]>
maxdebayser
maxdebayser reviewed Oct 2, 2025
View reviewed changes
vllm_spyre/model_executor/model_loader/spyre.py Outdated
self.kv_cache_specs['head_dim'] = self.config.hidden_size // \
self.config.num_attention_heads

# *** ALERT *** Granite 2b hack for AIU Compiler
Copy link
Collaborator

@maxdebayser maxdebayser Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know if this is a temporary hack? Then perhaps we can add a compatibility test

Copy link
Collaborator Author

@yannicks1 yannicks1 Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will see how they implement it properly in fms. I do just mirror their strategy here... (very much work in progress)

@yannicks1
Copy link
Collaborator Author

yannicks1 commented Oct 15, 2025

bot:test

@yannicks1 yannicks1 changed the title [WIP] Granite 2b & 3b support Granite4 2b & 3b support Oct 15, 2025
@yannicks1
Copy link
Collaborator Author

yannicks1 commented Oct 15, 2025

bot:test

@yannicks1 yannicks1 marked this pull request as ready for review October 15, 2025 12:31
maxdebayser
maxdebayser approved these changes Oct 15, 2025
View reviewed changes
Copy link
Collaborator

@maxdebayser maxdebayser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I suppose we can add the model to the know supported configurations once we have PELE test coverage.

cc: @ckadner , @joerunde , @jfialho

@yannicks1 yannicks1 merged commit de0e3d2 into main Oct 16, 2025
20 checks passed
@yannicks1 yannicks1 deleted the ysc-support-granite-2b-3b branch October 16, 2025 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxdebayser maxdebayser maxdebayser approved these changes

@tdoublep tdoublep Awaiting requested review from tdoublep tdoublep is a code owner

@nikolaospapandreou nikolaospapandreou Awaiting requested review from nikolaospapandreou nikolaospapandreou is a code owner

@sducouedic sducouedic Awaiting requested review from sducouedic sducouedic is a code owner

@joerunde joerunde Awaiting requested review from joerunde

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yannicks1 @maxdebayser